Apparatuses and methods for three-dimensional imaging of an object

ABSTRACT

Various embodiments include an apparatus including a phase mask and circuitry. The phase mask is configured and arranged with optics in an optical path to modify a shape of light, passed from an object. The shape modification characterizes the light as having two lobes with a lateral distance that changes along a line, having a first orientation, as a function of an axial proximity of the object to a focal plane, and with the line having a different orientation depending on whether the object is above or below the focal plane. The circuitry is configured and arranged to generate a three-dimensional image from light detected at the image plane, by using the modified shape to provide depth-based characteristics of the object.

FEDERALLY-SPONSORED RESEARCH AND DEVELOPMENT

This invention was made with Government support under contract GM085437awarded by the National Institutes of Health. The Government has certainrights in the invention.

BACKGROUND

Understanding of living cells is beneficial for a variety of researchand development. In order to understand the complex machinery at workwithin living cells, the position of individual biomolecules ismeasured. For example, single-particle tracking (SPT), in which thetrajectory of a moving individual molecular label, quantum dot, ornanoparticle is determined from a series of images, provides a valuabletool for a wide range of biological applications. Information inferredfrom the extracted particle trajectory sheds light on physicalproperties such as particle size, conformation, and the localenvironment, because observing the motion of single particles directlyunmasks nanoscale behavior such as diffusion, directed motion, binding,or anisotropy. If only two-dimensional trajectories are recorded,information is missed.

These and other matters have presented challenges to three-dimensionalimaging of particles, for a variety of applications.

SUMMARY

The present invention is directed to overcoming the above-mentionedchallenges and others related to the types of devices discussed aboveand in other implementations. The present invention is exemplified in anumber of implementations and applications, some of which are summarizedbelow as examples.

Various embodiments in accordance with the present disclosuresimultaneously measure all of these physical parameters, with minimalmodification of a conventional microscope. For example, in someembodiments, a 4f optical processing circuitry is added to the camera.In addition, such apparatus embodiments include a relatively simple andcheap light-sheet microscope.

Various aspects of the present disclosure include an apparatuscomprising a phase mask and circuitry. The phase mask is arranged withoptics in an optical path to modify a shape of light, passed from anobject. For example, the shape modification characterizes the light ashaving two lobes with a lateral distance that changes along a line,having a first orientation, as a function of an axial proximity of theobject to the focal plane, and the line having a different orientationdepending on whether the object is above or below a focal plane. Invarious specific aspects, the shape of light is referred to as a“tetrapod point spread function.” The circuitry generates athree-dimensional image from the light by using the modified shape toprovide depth-based characteristics of the object.

Other related aspects of the present disclosure include a methodcomprising providing optics and a phase mask in an optical path. Lightis passed through the optical path to circuitry where the light isdetectable. The circuitry encodes an axial position of an observedobject based on the detected light. For example, the axial position ofthe observed object is encoded by modifying a point-spread-function(PSF) at the circuitry using one or more parameterized phase masks. Theone or more parameterized phase masks are optimized, in various specificaspects, for a target depth-of-field range for an imaging scenario. Forexample, the PSF in various specific aspects is used for a targetdepth-of-field of up to 20 microns.

Various more specific aspects of the present disclosure include anapparatus comprising an optical path and circuitry. The optical pathincludes an imaging circuit, optics, and a phase mask. The imagingcircuit is at an image plane in the optical path for detecting thelight. The optics pass light from an object toward the image plane. And,the phase mask is arranged with the optics to modify a shape of lightpassing along the optical path, passed from the objects. The lightpassing along the optical path is modified to create a PSF. For example,the shape modification characterizes the light as having two lobes witha lateral distance that changes along a line, having a firstorientation, as a function of an axial proximity of the object to thefocal plane, and the line having a different orientation depending onwhether the object is above or below a focal plane. The circuitgenerates a three-dimensional image from the light detected by using themodified shape to provide depth-based characteristics of the object.

DESCRIPTION OF THE FIGURES

Various example embodiments may be more completely understood inconsideration of the following detailed description in connection withthe accompanying drawings, in which:

FIGS. 1A-1B illustrate example apparatuses in accordance with variousembodiments;

FIG. 1C illustrates an example of a modified shape of light inaccordance with various embodiments;

FIG. 2 illustrates an example of an apparatus in accordance with variousembodiments;

FIGS. 3A-3D illustrate an example of a phase mask optimized for a depthof field range of 6 um in accordance with various embodiments;

FIGS. 4A-4D illustrate an example of a phase mask optimized for a depthof field range of 20 um in accordance with various embodiments;

FIG. 5 illustrates an example microfluidic apparatus, in accordance withvarious embodiments;

FIGS. 6A-6B illustrate examples of a microscope apparatus, in accordancewith various embodiments;

FIGS. 7A-7B illustrate examples of a light sheet microscope, inaccordance with various embodiments;

FIGS. 8A-8D illustrate examples of depth based characteristicsdetermined using an apparatus, in accordance with various embodiments;

FIGS. 9A-9B illustrate examples of a three dimensional image generatedusing an apparatus, in accordance with various embodiments;

FIGS. 10A-10B illustrate an example of tracking an object inthree-dimensions using an apparatus, in accordance with variousembodiments;

FIGS. 11A-11B illustrate examples of a tetrapod point spread function indifferent z-ranges, in accordance with various embodiments;

FIG. 12 illustrates an example three-dimensional tetrapod point spreadfunction, in accordance with various embodiments;

FIG. 13 illustrates an example precision of a tetrapod point spreadfunction, in accordance with various embodiments;

FIGS. 14A-14C illustrate examples of a mean-squared-displacement curveof a tetrapod point spread function, in accordance with variousembodiments;

FIGS. 15A-15C illustrate examples of three-dimensional localization ofan object using a tetrapod point spread function, in accordance withvarious embodiments;

FIGS. 16A-16D illustrate an example of three-dimensional tracking ofobjects using a tetrapod point spread function, in accordance withvarious embodiments;

FIGS. 17A-17D illustrate an example of three-dimensional tracking ofobjects using a tetrapod point spread function, in accordance withvarious embodiments;

FIG. 18 illustrates an example of three-dimensional tracking of anobject using a tetrapod point spread function, in accordance withvarious embodiments; and

FIGS. 19A-19C illustrate an example of two phase masks optimized for 6um used for two different wavelengths in accordance with variousembodiments.

While various embodiments discussed herein are amenable to modificationsand alternative forms, aspects thereof have been shown by way of examplein the drawings and will be described in detail. It should beunderstood, however, that the intention is not to limit the invention tothe particular embodiments described. On the contrary, the intention isto cover all modifications, equivalents, and alternatives falling withinthe scope of the disclosure including aspects defined in the claims. Inaddition, the term “example” as used throughout this application is onlyby way of illustration, and not limitation.

DETAILED DESCRIPTION

Aspects of the present disclosure are believed to be applicable to avariety of different types of apparatuses, systems and methods involvinga phase mask that modifies a shape of light passed from an object andcircuitry to generate a three-dimensional image by using the modifiedshape on a two-dimensional detector to provide depth-basedcharacteristics of the object. In certain implementations, aspects ofthe present disclosure have been shown to be beneficial when used in thecontext of optical microscopy or with point spread functions that varybased on a depth range. While the present invention is not necessarilylimited to such applications, various aspects of the invention may beappreciated through a discussion of various examples using this context.

Accordingly, in the following description various specific details areset forth to describe specific examples presented herein. It should beapparent to one skilled in the art, however, that one or more otherexamples and/or variations of these examples may be practiced withoutall the specific details given below. In other instances, well knownfeatures have not been described in detail so as not to obscure thedescription of the examples herein. For ease of illustration, the samereference numerals may be used in different diagrams to refer to thesame elements or additional instances of the same element.

According to various example embodiments, aspects of the presentdisclosure are directed to three-dimensional (3D) localization ofindividual objects over a customizable depth range in opticalmicroscopy. In some embodiments, a conventional microscope is modifiedand the shape of a point-spread-function (PSF) is used to encode theaxial (depth) position of an observed object (e.g., a particle). The PSFis modified by Fourier plane processing using one of a set ofparameterized phase masks, which is optimized for a depth-of-field rangefor the imaging scenario. An object, as used herein, includes anemitter, such as a particle, a molecule, a cell, a quantum dot, ananoparticle, etc.

SPT techniques are typically based on frame-by-frame localization of theparticle. Namely, a series of time-sequential images (frames) arecaptured using a microscope, and each frame is analyzed to yield thecurrent position of the particle. In some applications, the extractedpositions are in two dimensions (2D), comprising lateral, or x,ycoordinates. The noisy and pixelated 2D detector image of the particleis analyzed, e.g., by centroid or Gaussian fitting, to yield theestimated x, y coordinates of the particle. However, as many samples ofinterest are inherently three-dimensional (3D), the full physicalbehavior of the tracked object is revealed by analyzing its 3Dtrajectory. The 3D trajectory of a moving particle can be extracted inseveral ways. For example, a particle can be followed by using afeedback control loop based on moving a 3D piezo stage according to thereading of several detectors (e.g., photodiodes). While providing a veryprecise temporal and spatial trajectory, this method is inherentlylimited to tracking a single particle.

Alternatively, scanning methods, such as confocal microscopy, areimplemented, in which an illumination beam or the focal point of themicroscope (or both) are scanned over time in three dimensions to yielda 3D image of the object. Scanning methods are limited in their temporalresolution, since at a given time only a small region is being imaged.In order to simultaneously track several particles in 3D, a scan-freewidefield approach can be used.

In accordance with various embodiments, 3D microscopic localization ofpoint-like light objects is generated using wide-field microscopy. Whena point-like (e.g., sub-wavelength) source of light is positioned at thefocal plane of a microscope, the image that is detected on the imagingcircuitry, such as a camera and/or a detector, is known as the PSF ofthe microscope. A conventional microscope's PSF (e.g., essentially around spot) is used for imaging a two-dimensional (2D) ‘slice’ of aspecimen, and for 2D (x,y) transverse localization of an object withinthat slice. That is, by fitting the shape of the spot with a 2D functionsuch as a centroid, Gaussian, or Airy function, in some instances, theposition of the object is detected with precision (a process termedsuper-localization). However, objects that are a small distance above orbelow the microscope's focal plane can appear blurry, and furthermore,their depth (or axial distance from the focal plane) is difficult todetermine from their measured image. In accordance with variousembodiments, 3D (x,y, and z) position information is obtained, even whenan object is above or below the focal plane. Using a phase mask, anadditional module is installed on a conventional microscope to solve theblur and depth issues. Instead of a point of light forming a single‘spot’ on the camera, light passing through the phase mask forms a shapeon the camera that looks different as a function of the object anddistance from the focal plane (or amount of defocus).

Certain embodiments in accordance with the present disclosure includeuse of an optimization technique including PSFs with impressive depthranges. Surprisingly, for a given optical system (e.g., with limitationsdefined by an objective lens), depth ranges are realized, for anapplication, far beyond previously-known range limits of 2-3 um. As aspecific example, using a phase mask optimized for a particular depthrange, super-localization over a customizable depth range is performedup to 20 um using a 1.4 numerical aperture (NA) objective lens. However,embodiments are not so limited and the depth range in variousembodiments is greater than 20 um. The depth range, for example, is afunction of the NA objective lens and the light emitted by the object.In various embodiments, the PSF is used for 3D super-localization andtracking, as well as for 3D super-resolution imaging in biologicalsamples, since this is an applicable depth range used for observing the3D extent of a mammalian cell.

Certain PSFs, in accordance with the present disclosure may be referredto as tetrapod PSFs, due to the shape they trace out in 3D space, as afunction of the emitter position (the position of the object). In anumber of embodiments, the modified shape characterizes the light ashaving two lobes with a lateral distance that changes along a line,having a first orientation, as a function an axial proximity of theobject to the focal plane, and the line having a different orientationdepending on whether the object is above or below a focal plane. Forexample, the different orientation of the line as compared to the firstorientation, in various embodiments, includes a lateral turn of the linefrom the first orientation to the different orientation, such as a 90degree or 60 degree lateral turn. This shape has lines from the centerof a tetrahedron to the vertices, or like a methane molecule. The PSF iscomposed of two lobes, where their lateral distance from one another andorientation are indicative of the z position of the object. Above thefocal plane, the two lobes are oriented along a first line, and belowthe focal plane the two lobes are oriented along a second line that isdifferently orientated than the first line (e.g., perpendicular to thefirst line). For example, the modified shape is created, in variousembodiments, by decreasing the lateral distance (e.g., moving together)of the two lobes along the first line when the object is above the focalplane and is closer to the focal plane (e.g., moving closer), turningthe two lobes laterally, such as 90 degrees, and increasing the lateraldistance (e.g., moving apart) of the two lobes another along the secondline when the object is below the focal plane and is further away fromthe focal plane (e.g., moving away).

Emitter (e.g., object) localization can be optimally performed usingmaximum likelihood estimation, based on a numerical or experimentallyobtained imaging model. However, other localization methods can be used.While other methods for 3D imaging can be used, such methods usescanning (e.g. confocal), in which temporal resolution is compromised,or parallelizing the imaging system (multi-focal imaging), whichcomplicates the implementation. Embodiments in accordance with thepresent disclosure do not use a scan or parallelization technique, andinclude observation of multiple single emitters in a field at highprecision throughout depth ranges, such as discussed above.

In accordance with specific embodiments, aspects of the presentdisclosure involve 3D super-localization microscopy techniques. Suchtechniques can include tracking single biomolecules with fluorescentlabels inside a biological sample, and 3D analysis using other lightemitting objects such as quantum-dots or the scattered light from goldbeads or nano-rods. Additionally, various embodiments include use of amicrofluidic device to characterize flow in 3D. Embodiments inaccordance with the present disclosure mitigate background noise in themeasured image that is caused by fluorescent emitters that are outsidethe focal plane being optically excited, and therefore emit light (whichcontributes to background noise in the measured image). One method tomitigate background noise includes light-sheet microscopy (LSM). In LSM,only a narrow slice of the thick sample is illuminated at a given time,therefore only objects (e.g., emitters) within that slice are active(illuminating).

In various embodiments of the present disclosure, an LSM (e.g., arelatively simple LSM) is used in combination with a tetrapod PSF. Forexample, with a tetrapod PSF, depth information is encoded in the PSFshapes and the sample is illuminated in a descending angle relative tothe field of view. The z-slice illuminated by the LSM is not parallel tothe focal plane of the object, but rather, it is tilted by some angle.Due to the large depth range, PSFs in accordance with the presentdisclosure can accommodate an angle that is steep (tens of degrees).Therefore, imaging is performed all the way down to the substrate, andthe light sheet is scanned. The tetrapod PSF, as used herein, is not arotation of a shape of the passing light (e.g., relative to a centerline) as a function of the axial position of the object (as with aspiral and/or helix PSF). Such embodiments can be advantageouslyimplemented relative to previous LSM schemes. Such previous LSM schemescan be difficult to implement because imaging that is close to thebottom of the sample involves overlapping the illumination beam with theunderlying glass substrate, which distorts the beam and prevents theformation of an undistorted light-sheet illumination profile. ThereforeLSM techniques (Bessel beam methods, for example) are cumbersome,costly, or use stringent manufacturing constraints. In onedual-objective design based on 45 degree excitation and collectionobjectives, the imaging is constrained to using low numerical aperture(NA) objective lenses, limiting the photon collection efficiency andultimately reducing precision.

According to various example embodiments, aspects of the presentdisclosure are directed to an apparatus or method involving encoding anaxial (e.g., depth) position of an observed particle by modifying apoint-spread-function (PSF) using one or more parameterized phase masks.In various embodiments, each such parameterized phase masks areoptimized for a target depth-of-field range for an imaging scenario. Inspecific embodiments, the optics pass light from an object toward theimage plane and the phase mask. The phase mask is used to modify a shapeof light, passed from the object. The shape modification includes ashape of light as a function of an axial proximity of the object, suchas a tetrapod PSF. In various embodiments, the shape of light ischaracterized by having two lobes with a lateral distance that changesalong a line, having a first orientation, as a function of an axialproximity of the object to a focal plane, and with the line having adifferent orientation depending on whether the object is above or belowthe focal plane.

The circuitry infers depth information about objects that are imaged.For example, the circuitry can be configured to infer depth of portionsof the object based on the modified shape and a degree of blur, atetrapod point-spread function (PSF), a 3D shape of the object on theimage plane and a location of a portion of the object from which thelight is emitted, and/or a Zernike polynomial (and any combinationthereof). In some embodiments, the circuitry generates the 3D imagebased on a Zernike polynomial of at least a 3^(rd) order. In variousembodiments, the generated 3D image is indicative of respective depthsof portions of the object that are greater than 3 microns from oneanother.

The phase mask, in some embodiments, is a deformable mirror used to tunethe depth characteristic by deforming. For example, the phase mask tunesa depth characteristic to obtain light from the object at differentrespective depths. In some embodiments, the apparatus and/or method, asdescribed above, includes a tuning circuit used to tune the depthcharacteristic.

In a number of particular embodiments, an apparatus and/or method inaccordance with the present disclosure is used to track objects. Forexample, an apparatus and/or method is used to localize an object, tracklocations and/or movement of an object, track locations and/or movementof multiple objects simultaneously, and/or characterize flow in 3D in amicrofluidic device (and any combination thereof).

In embodiments involving single-molecule based super-resolutionmicroscopy, combining a tetrapod PSF with a tilted light-sheetmicroscope allows for depth measurements of individual fluorescingmolecules over a depth range that reaches or exceeds 20 um. This data isused to construct a 3D image of a large biological structure (e.g.,whole mammalian cell) with resolution surpassing the diffraction limitby an order of magnitude. In the context of single-particle trackingmicroscopy, the phase mask allows for the 3D position of individualsub-diffraction limited objects to be monitored. An apparatus inaccordance with various embodiments is used to track particles such asfluorescent molecules, quantum dots or the scattered light from goldbeads or nanorods.

In a number of embodiments, phase mask design parameters may be adjustedto deliver optimal performance for a given depth range. Thereby, thephase mask in accordance with the present disclosure is not as limitedin depth range as other depth estimation techniques. A moduleincorporating a phase mask, in accordance with various embodiments, isinstalled on an existing microscope (e.g., commercial microscope) in ashort period of time, such as less than thirty minutes. A phase mask canallow for a high numerical aperture (NA) implementation forlight-sheet-microscopy.

In accordance with various embodiments of the present disclosure, 3Dposition information is extracted from a single widefield 2D image, bymodifying the microscope's point spread function (PSF), namely, theimage which is detected when observing a point source. Examples of PSFalterations which are used for 3D tracking and imaging under biologicalconditions include astigmatism, the double-helix PSF, the corkscrew PSF,the bisected-pupil PSF, and an Airy-beam-based PSF, with applicablez-ranges of around 1-2 μm for astigmatism and the bisected pupil PSF,and around 3 μm for the double-helix, corkscrew, and Airy PSFs.

Embodiments in accordance with the present disclosure include generating(information-optimal) PSFs for 3D imaging based on numericallymaximizing the information content of the PSF. Surprisingly, theresulting PSF exhibits superior 3D localization precision over otherPSFs. Despite gradual improvements in PSF designs, other PSFs can belimited in terms of their applicable z-range. Currently, the z-range ofother PSF designs is limited to around 3 μm, posing a major limitationfor applications requiring ‘deep’ imaging. For example, the thickness ofa mammalian cell can be larger than 6 μm and in the case of cells grownon cell feeder layers or in 3D cell cultures, which are becomingincreasingly popular in the biological community, samples are muchthicker than 3 μm.

In various embodiments of the present disclosure, by utilizing theinformation maximization framework, a group or family of (tetrapod-type)PSFs are used for 3D localization over a depth range far larger than theapplicable depth ranges of other designs, such as optimized for rangesof 2-20 um. By setting the optimization parameters to correspond to thedesired depth range, specific PSFs yield 3D localization optimized overthe range. For example, in various embodiments, a tetrapod PSF can beoptimized for a 20 μm z-range, and as may be applicable toflow-profiling in a microfluidic channel. In other embodiments, such aPSF is optimized for a 6 μm z-range under biological conditions (e.g.,tracking single quantum-dot labeled lipid molecules diffusing in livemammalian cell membranes).

Turning now to the figures, FIG. 1A illustrates an example apparatus 109in accordance with various embodiments of the present disclosure. Insuch embodiments, the apparatus 109 is used to design a family of pointspread functions (PSFs). The family of PSFs allow for preciselocalization of nanoscale emitters in 3D over customizable axial (z)ranges of up to 20 μm, with a high numerical aperture objective lens.For example, axial (e.g., depth) position of an observed particle and/orother object is inferred by modifying a PSF using one or moreparameterized phase masks. In some embodiments, the apparatus is used toperform flow profiling in a microfluidic channel, and illustratesscan-free tracking of single quantum-dot-labeled phospholipid moleculeson the surface of living, thick mammalian cells

The apparatus 109 includes a phase mask 103 and circuitry 107. The phasemask 103 is arranged with optics 101 in an optical path. For example,the optical path is from the optics 101 to the phase mask 103 to thecircuitry 107. The optics pass light from an object toward the imageplane. For example, the optics can include the various lenses, asillustrated by FIGS. 1B and 2. The phase mask 103 modifies a shape ofthe light based on a distance of respective portions of the object fromthe image plane. For example, the modified shape characterizes the lightas having two lobes with a lateral distance that changes along a line,having a first orientation, as a function of an axial proximity of theobject to the focal plane, and the line having a different orientationdepending on whether the object is above or below a focal plane. In somespecific embodiments, the lateral distance decreases as a function ofthe axial proximity of the object to the focal plane. For example, asthe object gets closer to the focal plane, the lateral distancedecreases. Similarly, as the object gets farther away from the focalplane, the lateral distance increases. Further, in various relatedembodiments, the two lobes are oriented along a first line with alateral distance that changes as the function of the axial proximity ofthe object to the focal plane and when the object is above the focalplane. As the object approaches and/or is in the focal plane, the twolobes turn orientations, such as 90 degrees. And, the two lobes areoriented along a second line with another lateral distance that changesas the function of the axial proximity of the object to the focal planeand when the object is below the focal plane. The first line beingparallel, or at another angle based on the turn of the lobes, to thesecond line.

The modification of light by the phase mask 103, in various embodiments,creates a modified PSF. For example, the PSF includes a tetrapod PSF, asfurther illustrated by FIG. 1C. That is, the modified shape includes atetrahedral shape as a function of the axial position, as furtherdescribed herein. The object, as used herein, corresponds to and/orincludes an emitter, such as a particle, a molecule, a cell, a quantumdot, a nanoparticle, etc. In various embodiments, the object is locatedin a sample and/or is labeled using a color. Various figures of thepresent disclosure illustrate a phase mask pattern, such as an inputvoltage pattern, as the phase mask. As may be appreciated, the actualphase mask is patterned with the illustrated phase mask pattern.

The phase mask 103, in various embodiments, is placed in the Fourierplane to modify light in the optical path. For example, the phase mask103 modifies a shape of the light by redirecting and modifying the lightpassing along the optical path to create a tetrapod PSF at the imageplane (e.g., the circuitry 107). The phase mask has a pattern thatincludes two peaks and two valleys, such as a two-dimensional saddlepoint function with two peaks and two valleys. The phase mask 103creates the tetrapod PSF, in various embodiments, by moving two lobestoward one another along a first line and when the object is above thefocal plane, turning the two lobes 90 degrees laterally, and moving thetwo lobes apart from one another along a second line that isperpendicular to the first line and when the object is below the focalplane. Accordingly, a feature of the tetrapod PSF is two lobes that varyin separation as a function of the object depth. The axis along whichthe lobes separate rotates 90 degrees or various other orientations,depending on whether the object is above or below the focal plane.

The circuitry 107 generates a 3D image from the light detected via thecircuitry 107. The circuitry 107 uses the modified shape (e.g., thetetrapod PSF) to provide depth-based characteristics of the object. Thedepth-based characteristics include the 3D position information (x, y,and z) and/or axial dimension (z). For example, the circuitry 107 infersdepth of portions of the object based upon the PSF. In variousembodiments, the circuitry 107 can include an imaging circuit at theimage plane (e.g., the final image plane) for detecting the light.

In various embodiments, the circuitry 107 generates the 3D image that isindicative of respect depths of portions of the object that are at least3 um from one another. For example, the circuitry 107 infers depth ofportions of the object based upon a 3D shape of the object on the imageplane and a location of a portion of the object from the emitted light.In various embodiments, the circuitry 107 infers the depth of portionsof the object based on Zernike polynomial, such as a Zernike polynomialof at least a 3rd order, as discussed further herein.

The circuitry 107, in accordance with a number of embodiments, encodesthe axial position of the object based on the tetrapod PSF created bythe phase mask 103. Encoding the axial position allows for the circuitry107 to localize an object, such as a particle, in 3D based on thetetrapod PSF. Further, locations of one or more objects in 3D aretracked by the circuitry 107 over a period of time and simultaneouslybased on the encoded axial position. For example, the circuitry 107characterizes flow of the one or more particles, such as in amicrofluidic device.

In various embodiments, the phase mask 103 tunes the depthcharacteristics to obtain light from the object at different respectivedepths. For example, the apparatus 109 includes a tuning circuit thatmanipulates the phase mask 103 to tune the depth characteristics. Insuch embodiments, the phase mask 103 includes a deformable mirrorconfigured to tune the depth characteristics.

The specific phase mask design (corresponding to a certain PSF from thetetrapod family) is dependent on the apparatus parameters, mainly on thedepth range. For different apparatus 109 parameters (magnification,numerical apertures, etc.), the phase mask 103 is optimized using anoptimization routine, as discussed further herein. For example, in someembodiments, the phase mask 103 yields slightly different phase maskpatterns. Related embodiments in accordance with the present disclosureutilize PSF engineering to provide optimized, high-precisionlocalization capability, for a large depth range. For example, such aphase mask design can yield a depth range of 2-20 um for a 1.4 NAobjective lens (with other parameter-set limitations, such asmagnification, background/signal levels, and noise issues). In someembodiments, the tetrapod PSF is tailored and optimized to a specificdepth range, which is dependent on and/or defined as a function of theinformation encoded in the PSF, as well as the NA objective lens and thelight emitted by the object. Surprisingly, it has been discovered, withsuch a tetrapod-type PSF, the information for a given amount of lightemitted by an object and for a given NA objective lens has asignificantly greater precision than other PSFs.

In some embodiments, a number of functions (e.g., equations andalgorithms) for specifying the exact design of a phase mask 103 (e.g.,parameterized phase mask) are used based on the system parameters of agiven imaging application. In addition, in various embodiments, thelocalization of an object given a measured image of the PSF is performedusing maximum-likelihood-estimation. In a number of embodiments, amodule (e.g., computer-readable code) is executed by the circuitry 107of the apparatus 109 to perform both of these actions, including theimaging model (as discussed further herein). At the same time, invarious embodiments, a set of phase masks are calculated to producetetrapod PSFs for various fixed z-ranges, and the expected shapes of thePSFs are provided as a library (e.g., stored using the circuitry 107).In accordance with various embodiments, no calculation is required bythe user except to perform fitting of the acquired images usinginterpolation of images from the library.

FIG. 1B illustrates an example apparatus in accordance with variousembodiments. As illustrated, the apparatus includes a modifiedmicroscope. The apparatus includes a phase mask 106 that is installed inan optical path (e.g., detection pathway) of the microscope. Phasemasks, in accordance with various embodiments, allow for precise 3Dlocalization of multiple objects (e.g., emitters) over a customizabledepth range. The customizable depth range, in some embodiments, is up to20 um for a 1.4 NA objective lens. The phase mask 106, in variousembodiments, is used to simultaneously track the location of multipleobjects at different depths, allowing scan-free high speed imaging. Thephase mask 106, in some embodiments, includes a mirror, such as adeformable mirror.

As illustrated, the apparatus includes an optical path that includesoptics 104, 108 that pass light from an object 112 from an object planetoward an image plane 110. A phase mask 106 is arranged with the optics104, 108 to modify the shape of light passed from the object 112. Forexample, in some embodiments, the phase mask 106 is positioned betweenthe objective lens 104 and a tube lens 108. As illustrated, the object112 in various embodiments is outside (e.g., above or below) a focalplane 102 of the apparatus. That is, the z-position of the object 112 isnot zero.

Such circuitry can be located at the image plane 110 for generating a 3Dimage from light detected at the image plane 110 using the modifiedshape to provide depth-based characteristics of the object 112. Thecircuitry, in various embodiments, includes imaging circuitry. Theimaging circuitry is circuitry commonly used with digital signal imageprocessing (such image circuitry includes, e.g., a charge-coupled device(CCD), image sensors based on field-effect technology such as CMOS imagesensors, and the like).

FIG. 1C illustrates example modifications of light by a phase mask, inaccordance with various embodiments. For example, the phase mask 100creates a tetrapod PSF 105 in various embodiments. In variousembodiments, the tetrapod PSF 105 includes a characterization of lighthaving two lobes with a lateral distance that changes along a line,having a first orientation, as a function of an axial proximity of theobject to the focal plane, and the line having a different orientationdepending on whether the object is above or below a focal plane. Asillustrated, the phase mask 100 modifies light to create a (tetrahedral)shape of light as a function of an axial position of the object thatresembles a methane molecule. In a number of embodiments, the shape hastwo lines from the center of the shape to two vertices oriented along afirst line when the object is above the focal plane and two lines fromthe center to two vertices oriented along a second line when the objectis below the focal plane. The second line is has a different orientationthan the first line, such as the second line being at an angle (e.g.,perpendicular) to the first line. A 3D rendering of the tetrapod PSF 105illustrates the intensity as a function of the object's axial positionaway from the focal plane (at z=0) which is coded by the shading (darkeras further above or below the focal plane). The graph 111 illustrateslocalizations of the object, as tracked using circuitry.

FIG. 2 illustrates an example apparatus in accordance with variousembodiments. As illustrated, the apparatus includes a modifiedmicroscope. The apparatus, in various embodiments, is a standard(inverted) microscope, augmented by a 4f optical processing system. ThePSF of the microscope is modified from that of a standard microscope bycontrolling the phase of the electromagnetic field in the Fourier planeof the 4f system using a phase mask 221. In various embodiments, the PSFof the standard microscope is modified by placing a phase mask 221 inthe Fourier plane of the microscope. The phase mask 221, in variousembodiments, includes a (dielectric) phase mask or a liquidcrystal-based spatial light modulator (SLM).

As illustrated by FIG. 2, the apparatus includes an optical path. Theoptical path is from the object 214 to the image plane 223. The object214, in various embodiments, is in a sample such as a biological sampleon a glass slip. The optical path includes optics 215, 216, 217,219,220, 222 configured to pass light from the object 214 toward the imageplane 223. The optics include one or more focusing lenses (e.g.,objective lens 215) and a light source to illuminate an object 214.Various other focusing lenses 216, 217,219, 222 focus the light, afterinteracting with the object 214, to the image plane 223 for detection bycircuitry. The optional polarizer 220 provides for polarizationselection in some embodiments. A phase mask 221 is located between anintermediate image plane 218 and the final image plane 223, in variousembodiments. For example, the phase mask 221 is placed in the back focalplane of optics 219, 220 and input focal plane of optics 222. Onceimplemented, an apparatus, consistent with that shown in FIG. 2, can beused to accomplish 3D imaging of an object and encode the axialposition. The phase mask 221, in some embodiments, is placed in betweenthe two optics 219/222. The mask, in accordance with the instantdisclosure, also serves the imaging functions of optics 219/222. Thephase mask, and as contemplated in other embodiments, is a hologram, acomputer-generated hologram, a diffractive optical element, a volumeoptical element, or the like. The mask may have one or both of amplitudeand phase components. The mask may be static or dynamic, based upon thescene being imaged, system requirements, or user requirements. The maskmay be implemented with a spatial light modulator which manipulates thephase and/or the amplitude of the light. Examples of such masks includethose constructed from liquid crystals or from micro-electro-mechanicalsystems. Further, a physical mask can also be fabricated, for example,by electron beam lithography, proportional reactive ion etching in SiO2,hot embossing in PMMA, gray level lithography, multistepphotolithography, or direct laser writing.

The light source in a number of embodiments includes a coherent lightsource. The coherent light source may include, for example, an Argon ionlaser operating at 488 or 514 nanometer (nm), or a diode laser emittingat 532 nm or 641 nm. Other lasers operating at various wavelengths mayalso be used as sources of coherent light. The light source may producemonochromatic or polychromatic light. The optics expand or reduce thelaser beam so that it illuminates a region of interest of the object214. These optics may also be supplemented by polarizers, waveplates, ordiffusers in order to manipulate the polarization or coherency of thelight illuminating the object. Other light sources that produceincoherent light, such as an arc lamp, may also be used. The focuselements may comprise, for example, coated achromatic lenses with 150 mmfocal length and an aperture of 50 mm. However, these focus elements maybe of different focal lengths in order to accomplish beam expansion orreduction. Various other lenses or optical components may also beincluded in order to focus light from the object onto the detector.

The circuitry, in various embodiments, encodes an axial position of theobserved object 214 by modifying a PSF at the circuitry using one ormore parameterized phase masks. For example, the parameterized phasemask 221 is optimized for a target depth-of-field range for an imagingscenario. The apparatus, for example, provides a target depth-of-fieldof greater than 2 micrometer (um) and up to at least 20 um. Encoding anaxial position, in various embodiments, includes localizing a particlein 3D based on the modified PSF (e.g., a tetrapod PSF), trackinglocations of multiple particles simultaneously based on the encodedaxial position, and/or characterizing flow in three-dimensions in amicrofluidic device

In various embodiments, the 3D image is generated using an image model.An example image model (e.g., an image formulation model) is built usinga function, such as:

I(u, v; x, y, z)∝|

{E(x′, y′; x, y, z)P(x′, y′)}|²,   (1)

where I(u, v) is the image, or the intensity in the image plane 223(e.g., camera plane), of a point source (e.g., the object) located atposition (x, y, z) in sample space, relative to the focal plane and theoptical axis (z). The field in the pupil plane, caused by the objectivelens 215 (e.g., point source), is denoted by E(x′, y′), and

represents the 2D spatial Fourier transform with appropriate coordinatescaling. The complex function P(x′, y′) is the pattern imposed in thepupil plane by a phase mask 221 (e.g., mask or an SLM).

Designing a PSF that allows for 3D localization over a large z-range,under high-background conditions (due to out-of-focus fluorescence orsample autofluorescence) can be challenging due to a number of demands.On one hand, the optics concentrate the light into a relatively smallregion throughout the applicable z-range in order to overcome backgroundnoise. On the other hand, the PSF contains Fisher information, i.e.,features which change as a function of z, such that its shape encodesthe z position of the emitter with high “recognizability” or“z-distinctness”, which then translates into high statisticallocalization precision when localizing under noisy conditions.

The precision of a given PSF is quantified by the Cramer Rao Lower Bound(CRLB). The CRLB is a mathematical quantity indicative of thesensitivity of a measurement to its underlying parameters. Morespecifically, the CRLB corresponds to the lowest possible variance inestimating these parameters with an unbiased estimator. In accordancewith various embodiments of the present disclosure, the measurement is anoisy, pixelated manifestation of the PSF (the 2D image), and theunderlying parameters are the 3D coordinates of the object (e.g.,emitter), as well as the brightness from the object expressed as totalsignal photons, and a background level of photons arising fromimperfections.

In various embodiments, given the apparatus parameters (such asmagnification, numerical aperture, background and signal levels, and a(Poisson) noise model), a numerical imaging model is built based onEq. 1. Such a model is used to find the Fourier phase pattern P(x′, y′)which yields the PSF with the lowest theoretical localization variance(equivalently—the lowest CRLB). The CRLB is related to the Fisherinformation matrix—more specifically, it is the inverse of the Fisherinformation. Therefore, the objective function being minimized is themean trace of the Fisher information matrix (corresponding to mean x,y,zCRLB) over a finite set of N unique z positions in a defined z-range.For example, the equation 2 includes a summary of the minimizationproblem:

$\begin{matrix}{\begin{matrix}{{Minimize}\; \text{:}} \\{{w.r.t.\text{:}}\mspace{14mu} {P\left( {x^{\prime},y^{\prime}} \right)}}\end{matrix}\mspace{14mu} \frac{1}{N}{\sum\limits_{j = 1}^{N}{{Trace}\left\{ F_{z_{j}}^{- 1} \right\}}}} & (2)\end{matrix}$

Wherein in Eq. (2) above, F_(z) _(j) is the 3-by-3 Fisher informationmatrix associated with the x-y-z localization precision of the PSF atthe j′th z position. This optimization is performed over a subset offunctions, e.g. Zernike polynomial. In various embodiments, the depth ofportions of the object 214 is inferred based on a Zernike polynomial.For example, circuitry generates the 3D image based on a Zernikepolynomial of at least a third order, as further described herein.

Performing optimization with different specified z-ranges, in someembodiments, yields different phase masks (and corresponding PSFs).However, the resulting PSFs share common characteristics. The commoncharacteristics include, for any tested z-range (from 2-20 μm), twodistinct lobes, with growing transverse distance between them as theemitter departs from the apparatus' focal plane. The orientation of thetwo lobes of the PSF, in some embodiments, is rotated by 90° above andbelow the focal plane. PSFs, in accordance with the present disclosure,are therefore referred to as tetrapod PSFs, due to the 3D tetrahedralshape they trace out as the object is moved in the z direction (e.g.,the axial direction).

FIGS. 3A-3D illustrate an example of a phase mask and correspondingtetrapod PSF optimized for a depth of field range of 6 um in accordancewith various embodiments. This phase mask 330 illustrated by FIG. 3A isoptimized to work in a (high) background scenario corresponding to livecell imaging conditions, such as, 3500 signal photons, and a meanbackground of 50 photons per pixels. The PSF measurements, in variousembodiments, are obtained by imaging a 200 nanometers (nm) fluorescentbead attached to an apparatus (e.g., microscope) cover slip, andscanning the apparatus objective such that the focal plane is above orbelow the bead. The resulting PSF 330 (e.g., a phase pattern), invarious embodiments, is used to concentrate the light into lobes andvary the PSF shape (e.g., shape of light) as a function of z positions.FIG. 3B illustrates modified PSF's 332-1, 332-2, 332-3, 332-4, 332-5 forvarious z positions as numerically calculated. FIG. 3C illustratesmeasured bead images for each of the z positions 333-1, 333-2, 333-3,333-4, 333-5 using the phase mask 330. FIG. 3D illustrates calculatedprecision (e.g., standard deviation for the parameter measurement) forthe z-position 334 and the x/y position 335.

FIGS. 4A-4D illustrate an example of a phase mask and correspondingtetrapod PSF for a depth of field range of 20 um in accordance withvarious embodiments. This phase mask 439 (e.g., a phase pattern),illustrated by FIG. 4A is optimized to work in a (high) backgroundscenario corresponding to live cell imaging conditions, such as, 3500signal photons, and a mean background of 50 photons per pixel. The PSFmeasurements, in various embodiments, are obtained by imaging a 200nanometers (nm) fluorescent bead attached to an apparatus (e.g.,microscope) cover slip, and scanning the apparatus objective such thatthe focal plane is above or below the bead. The resulting phase mask439, in various embodiments, is used to concentrate the light into lobesand vary the PSF shape (e.g., shape of light) as a function of z.Concentrating the light and varying the PSF shape is achieved due tooptimizing the objective function based on the CRLB. FIG. 4B illustratesmodified PSF's 440-1, 440-2, 440-3, 440-4, 440-5 for various z positionsas numerically calculated. FIG. 4C illustrates measured bead images foreach of the z positions 441-1, 441-2, 441-3, 441-4, 441-5 using thephase mask 439. FIG. 4D illustrates the calculated precision (e.g.,standard deviation of the parameter measurement) for the z-position 442and the x/y position 443.

In some embodiments, the calculated precision (e.g., standard deviation,defined as √{square root over (CRLB)}) for a signal of 3500 photons overa mean background of 50 photons per pixel is plotted, as illustrated byFIGS. 3D and 4D. According to the CRLB calculations, under particularsignal-to-noise conditions, the phase masks 330/439 illustrated by FIGS.3A and 4A exhibit a mean precision of 12 nm, 12 nm, 21 nm (29 nm, 29 nm,53 nm) in estimating x, y and z, respectively, using the 6 μm PSF (20 μmPSF). For more information on the tetrapod PSF optimization at differentz ranges, see Appendix C of the underlying provisional application,which is fully incorporated herein by reference.

That is, FIGS. 3A and 4A illustrate tetrapod masks, optimized forz-ranges of 6 μm and 20 μm. For example, FIGS. 3A and 4A illustrate 6 μmand 20 μm Tetrapod phase mask 330/439 patterns. FIGS. 3B and 4Billustrate numerical PSF calculation for various z position, and FIGS.3C and 4C illustrate measured bead images, each image normalized bymaximum intensity. FIGS. 3D and 4D illustrate numerically calculatedprecision, defined as √{square root over (CRLB)} for x, y and zdetermination, using 3500 signal photons on a background of 50 meanphotons per pixel.

FIG. 5 illustrates an example microfluidic apparatus 545, in accordancewith various embodiments. In various embodiments, an optical apparatusincluding a phase mask (e.g., a 20 μm Tetrapod mask 439 as illustratedby FIG. 4A) is used for flow profiling in a microfluidic channel. Themicrofluidic device 545 is useful for obtaining various measurements ofinterest, ranging from molecular diffusion coefficients or pH, andspanning 3D vascular modeling to inexpensive clinical diagnosticapplications. The use of PSF engineering provides a scan-free (andprecise) method for 3D flow profiling in such apparatuses.

In some embodiments, a laminar flow regime is analyzed. For general andspecific information on a laminar flow region, reference is made toBatchelor, G. K. An introduction to fluid dynamics; Cambridge universitypress: 2000, which is hereby fully incorporated by reference. Water witha low concentration (around 0.5 pM) of fluorescent beads is flowing549/550 through a glass microfluidic channel with a semi-circularcross-section (50±8 μm (width)×20±3 μm (height) near the center of thechannel). A 641 nm laser illuminates the sample, the widefieldfluorescence signal from the flowing beads is recorded, and a video istaken (5 millisecond (ms) exposures at 20 Hertz). The beads arelocalized as they flow, and the profile of the flow is obtained byanalyzing their trajectories, a technique calledparticle-image-velocimetry (PIV). 3D localization of each bead in eachframe is achieved using maximum-likelihood estimation based on fittingeach image to a numerical model of the PSF and taking into accountobjective defocus and refractive index mismatch between sample andmounting medium. For general and specific information about PIV,reference is made to Adrian, R. J.; Westerweel, J. Particle imagevelocimetry; Cambridge University Press: 2011; Vol. 30 and Cierpka, C.;Kähler, C. Journal of visualization 2012, 1, 1-31, both of which arehereby fully incorporated by reference.

The microfluidic device 545 includes a microfluidic channel setup withtwo beads 548, 547. Water with fluorescent beads (200 nm diameter, 625nm absorption/645 nm emission) is flowing through a micro-channel,placed on top of a microscope objective of an inverted microscope. Asthe beads 547, 548 flow, they are excited by a laser (641 nm), and theirfluorescence signal is captured.

FIGS. 6A-6B illustrate examples of a microscope apparatus, in accordancewith various embodiments. As illustrated by FIG. 6A the apparatusincludes various lenses 684, 685, 686 in an optical path between theobject 683 and the image plane. A phase mask 687 is located in theFourier Plane between two Fourier Transform lenses 685, 686.

FIG. 6B illustrates an example deformable mirror. In variousembodiments, the phase mask is a deformable mirror used to tune thedepth characteristic by deforming. For example, the phase mask tunes adepth characteristic to obtain light from the object at differentrespective depths. In some embodiments, the apparatus includes a tuningcircuit used to tune the depth characteristic.

A deformable mirror includes a mirror face-sheet 691 that is attached toan array of posts and an actuator array 689. For example, each post iscentered an actuator array 689. The actuator array 689 includes aflexible cantilever that is suspended over an actuator electrode 690.Further, the entire mirror face-sheet 691 and actuator array 690 isfabricated on a silicon wafer, in various embodiments.

FIGS. 7A-7B illustrate examples of a light sheet microscope, inaccordance with various embodiments. In various embodiments, theapparatus includes a light sheet microscope (LSM) as illustrated by FIG.7A. The LSM, at any given time, illuminates a slice of a sample and/oran object, around 2 um thick, by a sheet of light. A LSM, in variousembodiments, utilizes a tilted illumination 795 relative to the focalplane. For example, optics 793 illuminate 795 a slice of the sample 796and/or the object at a time. Thereby, the optics pass a sheet of lightthrough the sample 796 and/or the object via tilted illumination of thesample 796 and/or the object relative to the image plane. A scanningmirror 794, in various embodiments, is utilized to adjust the axialheight of the light sheet. Light emitted 799 from the slice of thesample is detected using circuitry 798, such as imaging circuitry. FIG.7B illustrates an example of illuminating 795 a slice 797 of a sample796 at a time using an LSM, such that only portions of the sample 796are illuminated at a given time.

In various embodiments of the present disclosure, an LSM (e.g., arelatively simple LSM) is used because the depth information is alreadyencoded in the PSF shapes and the sample 797 is illuminated in adescending angle relative to the field of view. For example, the z-sliceilluminated by the LSM is not parallel to the focal plane of theobjective, but rather, it is tilted by some angle. Due to the largedepth range, PSFs in accordance with the present disclosure provide anangle that is steep (tens of degrees). Therefore, imaging is performedall the way down to the substrate, and the light sheet is scanned in theaxial direction to sequentially illuminate the sample.

More Specific/Experimental Embodiments

FIGS. 8A-8D illustrate examples of depth based characteristicsdetermined using an apparatus, such as a microfluidic device and amodified microscope, in accordance with various embodiments. FIG. 8Ashows an example of a raw-data frame, with three beads at different x,y, and z positions simultaneously seen. By accumulating many frames(around 16000), the mean flow velocity as a function of x, y and z iscalculated. The example raw frame, of FIG. 8A, shows three emitters atdifferent x, y, z positions, flowing in the x direction.

FIG. 8B shows the y-z profile of the flow (which is in the x direction),whereas FIG. 8C and FIG. 8D show 1D cross-sections near the center ofthe channel. The x velocity profiles 861 are (reasonably) parabolic,while the mean y and z 862 are around 0. This fits well with a laminarflow model, assuming no slip conditions, where the Reynolds number is Re≈ 4·10⁻⁴. For example, FIG. 8B illustrates derived two-dimensional meanx-velocity map, averaged over x (y-z cross-section). FIGS. 8C and 8Dillustrate one-dimensional slices from FIG. 8B, showing mean x, y and zvelocities. As predicted by a laminar flow model, the mean x velocity861 has a parabolic profile, whereas they and z velocities 862 arenegligible by comparison.

In some embodiments, various quantities of interest are obtained by aquantitative analysis of the measured bead trajectories. For example, byanalyzing mean-squared-displacement (MSD) curves in they and zdirections (e.g., orthogonal to the flow), a mean diffusion coefficientof 1.20±0.13 (1.24±0.19) μm²/sec in they (z) direction is inferred. Thiscompares well with the theoretical value given by theEinstein-Smoluchowski relation for a 200 nm spherical diffuser in waterof 1.08±0.03 μm²/se. And, from the MSD curve intercepts, thelocalization precision is approximated. The resulting derived precisionsare 76 nm (87 nm) in they (z) estimation, in some embodiments.

To generate a 3D velocity profile, the microfluidic channel is imagedunder the input facet where the bead solution enters the microfluidicchannel. The beads are then imaged as they enter the channel, therebyexhibiting considerable flow also in the z direction.

FIGS. 9A-9B illustrate examples of a three dimensional image generatedusing an apparatus, in accordance with various embodiments. For example,FIGS. 9A-9B show the resulting flow profile obtained using an apparatuscomprising the microfluidic device with a microscope and/or imagingdevice. FIG. 9A illustrates three-dimensional trajectories ofone-hundred beads, with shade coding normalized per trajectory start(dark=first frame in trajectory) to end (light=last frame intrajectory). A typical trajectory lasts around 1.5 seconds. FIG. 9Bshows an x-z cross-section of the flow, near the center of the channel(in y) as illustrated by the inset 960. The flow is profiled over around30 μm in z. The data is binned in 3×3×3 μm³ x-y-z bins, arrow lengthlinearly encodes velocity (longest arrow corresponds to 22.5 μm/sec).

In some embodiments, several factors contribute to localization error inthe described flow analysis. One factor is signal-to-noise ratio,determined by the finite number of signal photons relative to backgroundphotons. However, in a number of embodiments, the measured beads arebright (number of signal photons per frame on the order of around100,000), and the background is low (a few photons per pixel) such thatthis is not a major contributor to the localization error. Motion bluris another cause for localization error. However, in some embodiments,this is not a major contributor to the localization error since theexposure time (5 ms) is short as compared to light velocities anddiffusion rates of the measured beads.

Another contribution to localization error comes from model mismatch.Model mismatch occurs when the model to which each measured PSF is fitdeviates from the actual modified PSF. This is partly because ofaberrations in the optical apparatus, and because of aberrations relatedto refractive index mismatch. The PSF of a point source (bead) in wateris somewhat different from the PSF of a bead on a cover-slip, andtherefore difficult to calibrate. In accordance with variousembodiments, the imaging model does include the effect of refractiveindex mismatch. The use of sophisticated numerical models and possiblycalibration methods, decreases the localization error that accompaniesthese kinds of measurements.

FIGS. 10A-10B illustrate an example of tracking an object in 3D using anapparatus, in accordance with various embodiments. For example, thetracking includes 3D tracking of a Quantum dot-labeledphosphoethanolamine (PE) lipid (e.g.,1070-1, 1070-2) on the surface of alive HeLa cell.

FIG. 10A, for example, illustrates a brightfield impression withoverlayed fluorescence channel (one 50 ms frame), showing a signal froma quantum-dot-labeled PE lipid. The scale bar, as illustrated, includes10 μm. FIG. 10B illustrates an inferred 3D trajectory 1074 as a functionof time (greater than 50 seconds total), shade-coded with timeprogression 1075, and planar projections of the motion shown in gray oneach of the bounding surfaces. The total motion over an axial range ofgreater than 5.5 μm is mapped. A maximum-likelihood estimation on aframe-by-frame basis produces the trajectory, in various embodiments.

In some embodiments, PSF optimization is used to analyze biologicalphenomena by performing 3D tracking of nanoscale objects using anoptical apparatus. For example, a phase mask is optimized for a 6 μmz-range to track the diffusive motion of single lipid molecules in alive cell membrane. FIG. 10A shows an example frame from a trackinganalysis following the motion of a quantum-dot labeledphosphoethanolamine (PE) lipid 1070-1, 1070-2 on the surface of a livingHeLa cell. The extracted 3D trajectory is plotted as illustrated by FIG.10B. The mean number of detected signal photons per frame is around10,000, with a mean background of around 40 photons per pixel. Theprecision in this measurement is estimated to be 10 nm in the x-ycoordinates and 17 nm in the z coordinate. This is measured bylocalizing immobilized quantum dots on the surface of the sample'scover-slip and averaging the standard deviation in localization, forseveral defocus values.

In various embodiments, the 3D trajectory of the molecule tracked inFIGS. 10A-10B is constrained to an approximately spherical surface. Thesphere that the trajectory outlines is visible in a white light image1073, as illustrated by FIG. 10A, and can be a detached bleb from anearby cell, pressed against the cell membrane from the outside. Whenfitting the molecule's trajectory to a sphere, the radius correspondswith the value obtained from the white light image 1073. While theexample embodiment in FIGS. 10A-10B show a single tracked molecule,PSF-engineering tracking allows for simultaneous tracking of multipleemitters.

In various embodiments, the PSFs are applicable to single-moleculelocalization microscopy. In some embodiments, single fluorescent dyemolecules (Alexa Fluor 647) are immobilized on a cover-slip. Themolecules are excited and their fluorescence is measured, using a 6 μm(tetrapod) PSF. Each molecule's position is then localized repeatedly.This is repeated for various defocus values throughout a 7 μm z-range.For a mean number of around 6000 detected signal photons and around 38background photons per pixel, the mean statistical localizationprecision, namely the standard deviation of localizations, averaged overthe entire z-range, is 15 nm, 12 nm and 29 nm in x, y and z,respectively.

Various embodiments include an imaging modality based on optimizedtetrapod PSFs, capable of high-precision imaging throughout a tunableaxial range. For example, large-axial-range tracking in a microfluidicdevice is performed, tracking under biological conditions of a

Qdot-labeled molecule diffusing on the membrane surface of livemammalian cells, as well as single-fluorophore localization capabilitiesover a 7 μm axial range. Thereby, the tetrapod PSF is used to performhigh-precision, scan-free tracking of multiple emitters over anexceptionally large z-range.

As previously discussed, an imaging model is used by the circuitry. Theimaging model is based on scalar diffraction theory of light from apoint source (i.e. polarization effects are not included), which yieldsimulations results that match experimentally obtained data. For adescription of more detailed modeling considerations, refer to FIG. 2.

The optical model consists of a two-layer experimental system consistingof water (refractive index n₁=1.33), and glass/immersion oil (which havematched refractive index of n₂=1.518). Light from a single emitter inthe sample acquires a phase factor (i.e. defocus) determined by thedistance between the emitter and the interface separating layer 1 andlayer 2 (z₂) and the distance between the microscope focal plane and theinterface (z₂). An additional phase factor P is imposed by the phasemask—which, by virtue of the 4f system, is modeled as being locatedwithin the pupil of the apparatus objective. For a given emitter, theoverall phase ψ_(pupil) of light at a given point {x′, y′} within theapparatus pupil is given by the equation:

$\begin{matrix}{{\psi_{total}\left( {x^{\prime},y^{\prime}} \right)} = {{P\left( {x^{\prime},y^{\prime}} \right)}e^{{{ikn}_{1}z_{1}\sqrt{1 - x^{\prime \; 2} - y^{\prime \; 2}}} + {{ikn}_{2}z_{2}\sqrt{1 - {\frac{n_{1}}{n_{2}}{({x^{\prime \; 2} + y^{\prime \; 2}})}}}}}}} & (3)\end{matrix}$

Note that if the interface is between the focal plane and the emitter,the sign of z₂ is positive. If the interface is closer to the objectivelens than the focal plane, z₂ is negative. In this expression, acoordinate system is used that is normalized such that points along thecircle √{square root over (x′²+y′²)}=N.A./n₂ lie on the outer edge ofthe tetrapod phase mask. Due to the objective, the electric field oflight along the outer edge of the microscope pupil has a greateramplitude than light close to the center of the pupil. This amplitudefactor A_(pupil) is given by:

$\begin{matrix}{{A_{pupil}\left( {x^{\prime},y^{\prime}} \right)} = \left\{ \begin{matrix}\left( \frac{1}{1 - x^{\prime \; 2} - y^{\prime \; 2}} \right)^{\frac{1}{4}} & {{{if}\mspace{14mu} \sqrt{x^{\prime \; 2} + y^{\prime \; 2}}} \leq {n_{1}/n_{2}}} \\0 & {otherwise}\end{matrix} \right.} & (4)\end{matrix}$

Using the imaging model, the region of non-zero amplitude is limited topoints inside the circle √{square root over (x′²+y′²)}≤n₁/n₂, due to thefact that super-critical light, inhabiting the region n₁/n₂<√{squareroot over (x′²+y′²)}=N.A./n₂ is attenuated for objects (e.g., emitters)which are an appreciable distance (λ<z₁) from the interface. After lighthas propagated beyond the objective, paraxial approximations are valid.Specifically, the tube lens of the microscope performs an opticalFourier transform operation. The electric field present at a point (u,v) in the microscope's image plane, (E_(img)) is given by the formula:

E _(img)(u, v)=FT{A _(pupil)(x′, y′)ψ_(pupil)(x′, y′)}=FT{E(x′,y′;x,y,z)P(x′, y′)},   (5)

where E(x′, y′) is the Fourier-plane electric field mentioned in Eq. 1.The intensity within the image plane is then:

I _(img)(u, v)=E _(img)(u,v)E* _(img)(u, v).   (6)

Finally, object-space coordinates (x, y) are related to image-spacecoordinates (u, v) by a scaling factor M, the overall magnification ofthe microscope.

The optimization procedure is based on a CRLB minimization method. Theobjective function being minimized is the mean CRLB in x, y and z, overa predetermined z-range composed of N distinct z (depth) values. Invarious embodiments, the optimization is performed over the set of thefirst 55 Zernike polynomials, so that the sought solution is acoefficient vector c ϵ R^(N) with N=55. The mathematical optimizationproblem, solved using Matlab's fmincon function, using the ‘interiorpoint’ method, is therefore:

$\begin{matrix}{{\min\limits_{c}{\sum\limits_{{j = \hat{x}},\hat{y},\hat{z}}{\sum\limits_{z \in Z}\sqrt{\frac{1}{I_{jj}\left( {c,z} \right)}}}}},} & (7)\end{matrix}$

where, assuming additive Poisson noise and a constant background of β,the Fisher information matrix for a point source along the optical axisis given according to Eq. 8 by:

$\begin{matrix}{{I\left( {{c;0},0,z} \right)} = {\sum\limits_{k = 1}^{N_{p}}{\frac{1}{{\mu_{c,z}(k)} + \beta}\left( \frac{\partial{\mu_{c,z}(k)}}{\partial\theta} \right)^{T}{\left( \frac{\partial{\mu_{c,z}(k)}}{\partial\theta} \right).}}}} & (8)\end{matrix}$

Here, θ=(x, y, z) is the 3D position of the emitter, summation isperformed over the sum of image pixels N_(p), and μ_(c,z) is a model ofthe detected PSF for an emitter at z, including the total number ofsignal photons per frame, magnification and pixelation, for a PSFproduced by a Fourier-plane mask P(x′, y′) defined by:

P(x′, y′)=circ(r/R)·exp(iD _(zer) ·c)   (9)

where r=√{square root over (x′²+y′²)}, R is the radius of the pupilplane,

${{{circ}(\eta)} = \begin{Bmatrix}{1,} & {\eta < 1} \\{0,} & {\eta \geq 1}\end{Bmatrix}},$

and D_(zer) is the linear operator transforming the vector of Zernikecoefficients to the 2D phase pattern to be projected on the SLM. The SLMis discretized to a 256×256 grid, so that D_(zer) ϵ R²⁵⁶ ² ^(×55), whereeach column is a vector-stacked 2D image of the corresponding Zernikepolynomial.

A set of tetrapod PSFs with z-ranges throughout the 2-20 μm range isderived by running the optimization procedure iteratively. Starting witha design z-range of 2 μm, the procedure is run once to produce anoptimal PSF. Then, the output solution is used as an initial point foranother iteration, with a larger z-range of 4 μm. This iterative processis repeated, iteratively increasing the z-range by 2 μm each time, to afinal z-range of 20 μm.

FIGS. 11A-11B illustrate examples of a tetrapod PSF in differentz-ranges, in accordance with various embodiments. The design range(z-range) has an effect on the resulting optimal tetrapod mask asillustrated by FIG. 11B. For example, the larger the z-range, the moreextreme the phase 1181 peaks and valleys in the Tetrapod mask. FIG. 11Aillustrates the z-range 1180-1, 1180-2, 1180-3, 1180-4, 1180-5 and FIG.11B illustrates the resulting optimized tetrapod phase masks 1182-1,1182-2, 1182-3, 1182-4, 1182-5 designed for z-ranges varying from 4-20um.

FIG. 12 illustrates an example 3D rendering of a tetrapod PSF, inaccordance with various embodiments. For instance, the 3D rendering ofthe tetrapod PSF 1213 includes a 6 μm tetrapod PSF image plane intensityas a function of the emitter's axial position away from the focal plane(at z=0). The intensities making up the 3D shape are thresholded forvisibility. Three slices of the PSF are shown with no thresholding (atz=−3,0,3 μm), displaying the full dynamic range of intensity. The scalebar is 1 μm. As illustrated, the PSF is created by 2 lobes 1224-1,1224-2 moved along a first line 1225 above the focal plane, turned 90degrees, and moved along a second line 1226 below the focal plane. Thesecond line 1226 is perpendicular to the first line 1225. Accordingly,in some embodiments, the modified tetrahedral shape is acharacterization of light having two lobes with a lateral distance thatchanges along a line, having a first orientation, as a function of anaxial proximity of the object to the focal plane, and the line having adifferent orientation depending on whether the object is above or belowa focal plane (e.g., two lines from the center to two lobes 1224-1,1224-2 orientated along a first line 1225 when the object is above thefocal plane and two lines from the center to the two lobes orientedalong a second line 1226 when the object is below the focal plane, thesecond line 1226 being perpendicular to the first line 1225).

As the z-range of the tetrapod PSF is increased, the mean resulting CRLBthroughout a 20 μm range decreases (by definition of the optimizationproblem), and this is indicative of mean precision enhancement. However,the overall mean improvement comes at a (local) cost as the CRLB aroundthe focus is increased. This result is shown in FIG. 13, where thecalculated z-CRLB is plotted as a function of z position of the emitterfor different tetrapod mask designs covering z-ranges between 4 and 20μm.

The calculation is for 3500 signal photons and background of 50 photonsper pixel, and a wavelength of 670 nm. For optimal results, the tetrapodphase mask matches the range of the problem. For example, using a 20 μmPSF for tracking within a 6 μm thick sample yields sub-optimalprecision.

FIG. 13 illustrates an example precision of a tetrapod PSF, inaccordance with various embodiments. For example, the graph of FIG. 13illustrates tetrapod mask precision vs. z-range. For a signal photonnumber of 3500 and 50 mean background photons per pixel, the 20 μmtetrapod PSF exhibits theoretical z precision of around 50 nm throughouta 20 μm z-range, however this comes at the cost of poorer precision nearthe focal plane (see the inset). Therefore, tetrapod masks engineeredfor smaller z-ranges are more suitable for imaging thinner samples. Forexample, the precision is illustrated for a 20 um z-range 1327-1, a 16um z-range 1327-2, a 12 um z-range 1327-3, an 8 um z-range 1327-4, and a4 um z-range 1327-5.

For the flow profiling experimental embodiments, such as using amicrofluidic device, nano-pure water with 200 nm diameter fluorescentbeads (FluoroSphere 625/645, Life Technologies), at a concentration ofaround 0.2-0.5 pM (roughly several beads per field of view) is flowedthrough a glass microfluidic channel (Micronit 0.3 μL thin bottommicroreactor). The cross-section of the channel in the y-z plane isapproximately a semicircle, with a radius of around 20 μm.

Constant-pressure-driven flow is maintained using a syringe pump (11plus, Harvard Apparatus). Measurements, in various embodiments, areperformed on an inverted microscope system (Olympus IX-71, objectivelens 100×/NA 1.4 oil immersion UPlanSApo, Olympus) with custom widefieldlaser excitation (641 nm Coherent Cube) and equipped with an EMCCD imagesensor (iXon+, DU 897-E, Andor). The microfluidic channel is placed inthe inverted microscope system, on top of the oil immersion objective(See FIG. 5). The nominal focal plane of the objective for the laminarflow measurements is at z=12 μm (e.g., 12 μm inside the channel), andfor the 3D flow measurement it is at z=12 μm

FIGS. 14A-14C illustrate examples of a mean-squared-displacement curveof a tetrapod PSF of laminar flow, in accordance with variousembodiments. The mean 3D velocity as a function of x,y,z is calculatedby binning the 3D field of view into 2×2×2 μm³ voxels, and calculatingthe mean velocity of emitters within each bin. The mean velocity per binis defined as:

$\begin{matrix}{{v_{b}^{i} = {\frac{1}{N}{\sum\limits_{n = 1}^{N}v_{n}^{i}}}},} & (10)\end{matrix}$

where i=x, y, z , and v_(n) ^(i) is the instantaneous velocity ofparticle n out of the total N particles measured in bin b. Theinstantaneous velocity vector is defined as:

$\begin{matrix}{{{\overset{\_}{v}}_{n} = \frac{\left\lbrack {{x\left( {t + {dt}} \right)},{y\left( {t + {dt}} \right)},{z\left( {t + {dt}} \right)}} \right\rbrack - \left\lbrack {{x(t)},{y(t)},{z(t)}} \right\rbrack}{dt}},} & (11)\end{matrix}$

where [x(t), y(t), z(t)] corresponds to the position of the particle attime t, and dt is the time difference between consecutive frames inwhich the particle was localized.

For the laminar flow measurement, statistical analysis is performed inorder to extract the diffusion coefficient and to estimate localizationprecision. This is done by analyzing the mean squared displacementcurves (MSD), shown in FIGS. 14A-14C. Linear fits with a constant offsetterm are extracted from the first three points (optimal number of MSDpoints is determined according to), where the slope is proportional tothe diffusion coefficient and the intercept is a combination of staticlocalization precision and the effect of a finite exposure duration.

FIG. 14A illustrates the x-direction, FIG. 14B illustrates the ydirection, and FIG. 14C illustrates the z-direction MSD curves obtainedfrom central 8 um strip of a laminar flow measurement. The x-directionexhibits a parabolic profile, whereas the y and z-directions exhibitlinear behavior, as expected from directed motion in the x direction inaddition to 3D Brownian motion. Derived diffusion coefficient from y (z)direction=1.20±0.13 (1.24±0.19) μm²/sec, matching a calculated value bythe Einstein-Smoluchowski relation of 1.09 μm²/sec. The y (z)localization error, estimated from the y-intercept, and accounting forfinite exposure time (5 ms), is 75 (91) nm.

For experimental demonstrations of lipid tracking in the plasma membraneon the 3D cell surface, two different cell lines are used: 1.Plastic-adherent sub-clone of the PC12 cell line (rat adrenal glandpheochromocytoma, ATCC Number: CRL-1721.1) and 2. HeLa cells (humanepithelial). Cells are grown at 37 degrees Celsius (C.) in 5% CO₂, 95%relative humidity atmosphere in tissue-culture treated flasks (75 cm²,BD Biosciences) in standard cell culture medium (10% fetal bovine serum(FBS), 90% phenol red free high-glucose Dulbecco's Modified Eagle'sMedium (DMEM), both Gibco). Cells are grown to confluency and passagedevery 2-3 days with 1:5 dilution two times prior to experiments. Cellpassaging consists of a 5 minute incubation with 2 milliliter (ml) oftrypsin replacement (TrypLE Express, Gibco), addition of 8 ml cell mediato deactivate the enzyme, a brief centrifugation (1 minute, less than400 relative centrifugal force (rcf)), followed by media replacement toremove any residual trypsin before transferring cells to a new flask. Inpreparation for imaging after the third passage, cell suspensions areadded to microscopy coverslips in 6-well tissue culture plates (BDBiosciences). These coverslips (standard no. 1.5, Fisher) arepre-cleaned, ozone-treated and (for PC12 cells) coated with 500 ml of8.4 mg/ml fibronectin (EMD Biosciences) in 1× phosphate buffered saline(PBS) (pH 7.4) for 1 hour, and rinsed with 1×PBS before use. Cells areattached on slides within a few hours and began to settle.

In various embodiments, cells are bulk-loaded with the lipidbiotinyl-cap phosphoethanolamine (PE)(1,2-dioleoyl-sn-glycero-3-phosphoethanolamine-N-(cap biotinyl), AvantiPolar Lipids, 870273) by use of fatty-acid-free bovine serum albumin(BSA). A 100 micrograms (μg)/ml lipid stock solution in absolute ethanolis prepared and stored at 4 degrees C. Labeling of cells is carried outdirectly in a sample holder that holds a microscope coverslip belowsmall (few ml) liquid volumes. Cells are loaded by diluting the lipidstock to 0.1-1 μg/ml in 0.1% fatty-acid-free BSA (Sigma Aldrich, A8806)to form complexes in Dulbecco's phosphate buffered saline (D-PBS) with0.1 grams (g)/L CaCl2 and 0.1 g/L MgCl2 (Life Technologies, 14040) andincubating for 5 minutes at room temperature. Qdot labeling aresubsequently performed by three washes of cells in D-PBS, followed byblocking for non-specific binding in D-PBS with 1% BSA for 1-2 minutes.Cells are brought in contact with a 1 nanometer (nM) filtered solution(using a 0.2 μm syringe filter) of the Qdot-655 streptavidin conjugate(Life Technologies, Q10123MP) in D-PBS with 1% BSA, for 2 minutes atroom temperature. Further binding is blocked by addition of 100 μl of 1mM biotin and further incubation at room temperature for 2 minutes.Labeled cells are washed three times in D-PBS and imaged in warm(initially 37° C.) D-PBS supplemented with 1% BSA. The above procedureresults in approximately 1-5 Qdot-labeled PE molecules per cell, anappropriate density for tracking.

For all cell tracking experimental embodiments, a 488 nm excitationsource is used from an Ar+ ion laser, at a level of around 650microwatts (μW) at the sample. For fluorescence filtering, a 488 nmdichroic mirror is used, combined with a 680/30 band-pass emissionfilter. The nominal focal plane of the objective lens is set at z=4.5μm.

FIGS. 15A-15C illustrate examples of 3D localization of an object usinga tetrapod PSF, in accordance with various embodiments. To estimate thelocalization precision in the HeLa cell tracking experiment (e.g., FIGS.10A-10B), immobilized Qdots are repeatedly localized on the surface ofthe cover slip of the same sample for tracking, at different defocusvalues. The mean number of detected signal photons per frame is around10,000, with a mean background of around 40 photons per pixel. The meanprecision is 10 nm in the x-y coordinates and 17 nm in the z coordinate.

For example, FIGS. 15A-15C illustrate the Qdots stuck to the HeLa cellcover slip surface repeatedly localized, for 3 objective defocus values:−3 μm (1528), 0 μm (1529), and 3 μm (1536). The localization scatterplots and histograms (Gaussian fits shown in red) are displayed for thethree defocus values. Standard deviations are stated above thehistograms.

Additional tracking examples using a 6 μm Tetrapod mask are shown,featuring the 3D trajectories of multiple Qdot-labeled PE lipidsdiffusing in the plasma membrane of PC12 cells as illustrated by FIGS.16A-16D and FIGS. 17A-17D

FIGS. 16A-16D illustrate an example of 3D tracking of objects using atetrapod PSF, in accordance with various embodiments. More specifically,FIGS. 16A-16D illustrate 3D tracking of Qdot-labeled phosphoethanolamine(PE) lipids on the surface of living PC12 cells. FIG. 16A illustrates abright field transmission image 1637-1 (left) and corresponding area inthe fluorescence channel 1637-2 (right, one 50 microsecond (ms) frame),showing signals from two fluorescent PE lipids (labeled 1 and 2). FIG.16B illustrates examples of the evolving fluorescence image of lipid 2,indicating frame number in the image sequence of 1900 frames. Engineeredshape changes by the optimized pupil plane phase design directly informthe x, y and z of the single emitter. FIG. 16C illustrates inferred 3Dtrajectories 1638-1 as a function of time (>90 seconds total), codedwith time progression. The total diffusion over an axial range ofgreater than 3 μm is mapped for lipid 2, while simultaneously tracking,without any scanning, the position of lipid 1, which remains largelystationary at greater than 2 μm below lipid 2. Maximum-likelihoodestimation on a frame-by-frame basis with respect to a reference libraryof image shapes as a function of z produces the trajectory. Separatetrajectories 1638-2, 1638-3 of the two lipids are displayed below. FIG.16D illustrates error histograms for x 1646-1, y 1646-2, z 1648-3 asstandard deviation (std.) or statistical precision computed as thesquare root of the CRLB calculated from the slightly time-varying setsof recovered photon number, background and position of all localizations(both lipids) in the movie. Mean precision values are displayed. Scalebars are 5 μm (a) and 2 μm (b).

FIGS. 17A-17D illustrate an example of 3D tracking of objects using atetrapod PSF, in accordance with various embodiments. More specifically,FIGS. 17A-17D illustrate simultaneous 3D three-emitter tracking ofQdot-labeled phosphoethanolamine (PE) lipids on the surface of a livingPC12 cell. Tracking results for two lipids diffusing on the peripheralparts of a rounder cell in the settling phase combined with a third lessmobile lipid (greater than 100 second observation, 2038 frames). Thethree lipids combined explore around 5 μm in z direction. FIG. 17Aillustrates a brightfield impression 1752-1 (left) and correspondingarea in the fluorescence channel 1752-2 (right, one 50 ms frame),showing signals from the three fluorescent PE lipids (labeled 1, 2, and3). FIG. 17B illustrates inferred 3D trajectories 1753 (overview) andFIG. 17C illustrates detailed 3D trajectories 1754-1, 1754-2, 1754-3shown individually as a function of time (around 100 seconds total),coded with time progression. FIG. 17D illustrates error histograms for x1755-1, y 1755-2, z 1755-3 as standard deviation (std.) or statisticalprecision computed as the square root of the CRLB calculated from theslightly time-varying sets of recovered photon number, background andposition of all localizations (all three lipids) in the movie. Meanprecision values are shown. Scale bars are 5 μm.

FIG. 18 illustrates an example of 3D tracking of multiple objects usinga tetrapod PSF, in accordance with various embodiments. Morespecifically, FIG. 18 illustrates tracking of 15 single molecules thatare repeatedly localized (around 75 frames for each molecule), and thex, y, z statistical localization precision is reported as a function ofthe z position of the objective (defocus value). To demonstrate thesingle-fluorophore localization capabilities of a 6 μm Tetrapod mask,Alexa Fluor 647 molecules are immobilized in PVA 0.3% on a cleanmicroscope cover-slip (VWR No. 1.5, ozone-cleaned for 15 minutes).Single molecules are imaged (100 ms frames) and localized for variousdefocus values, throughout a 7 μm z-range. The mean signal number ofphotons is around 6000, and the mean background is around 38 photons perpixel. The mean precision, defined as the standard deviation over a meannumber of 75 independent localizations per molecule, was (15,12,29) nm,in (x, y, z) respectively.

FIGS. 19A-19C illustrate an example of two phase masks optimized for 6um used for two different wavelengths in accordance with variousembodiments. For example, FIG. 19A illustrates a phase mask 1944optimized for 6 um and corresponding PSFs 1956-1, 1956-2, 1956-3,1956-4, 1956-5 across a 6 um z-range. In various embodiments, the phasemask design is used to create multiple phase masks. For example, twophase masks are implemented using the phase mask 1944 pattern for twodifferent wavelengths (e.g., red and green colors) to localize objectsin a sample that are labeled with different colors.

FIG. 19B illustrates the phase mask (bottom) and resulting fluorescentmicrosphere measurements (top) for a first wavelength (e.g., green) andFIG. 19C illustrates another phase mask (bottom) and resultingfluorescent microsphere measurements (top) for a second wavelength(e.g., red) across a 4 um z-range. In various embodiments, the differentwavelengths are used to measure two different locations in a sample,such as DNA in a live yeast cells.

Various blocks, modules or other circuits may be implemented to carryout one or more of the operations and activities described herein and/orshown in the figures. In these contexts, a “block” (also sometimes“circuitry”, “circuit”, or “module”) is a circuit that carries out oneor more of these or related operations/activities (e.g., optimization,encoding an axial position, detect light, generate 3D image, ormanipulate a phase mask). For example, in certain of the above-discussedembodiments, one or more modules are discrete logic circuits orprogrammable logic circuits configured and arranged for implementingthese operations/activities. In certain embodiments, such a programmablecircuit is one or more computer circuits programmed to execute a set (orsets) of instructions (and/or configuration data). The instructions(and/or configuration data) can be in the form of firmware or softwarestored in and accessible from a memory (circuit). As an example, firstand second modules include a combination of a CPU hardware-based circuitand a set of instructions in the form of firmware, where the firstmodule includes a first CPU hardware circuit with one set ofinstructions and the second module includes a second CPU hardwarecircuit with another set of instructions. Also, although aspects andfeatures may in some cases be described in individual figures, it willbe appreciated that features from one figure or embodiment can becombined with features of another figure or embodiment even though thecombination is not explicitly shown or explicitly described as acombination.

Terms to exemplify orientation, such as upper/lower, left/right,top/bottom, above/below, and axial/lateral (as well as x, y, and z), maybe used herein to refer to relative positions of elements as shown inthe figures. It should be understood that the terminology is used fornotational convenience only and that in actual use the disclosedstructures may be oriented different from the orientation shown in thefigures. Thus, the terms should not be construed in a limiting manner.

Certain embodiments are directed to a computer program product (e.g.,nonvolatile memory device), which includes a machine orcomputer-readable medium having stored thereon instructions which may beexecuted by a computer (or other electronic device) to perform theseoperations/activities.

Various embodiments are implemented in accordance with the underlyingProvisional Application (Ser. No. 62/146,024) to which benefit isclaimed and which is fully incorporated herein by reference. Forinstance, embodiments herein and/or in the provisional application(including the appendices therein) may be combined in varying degrees(including wholly). Reference may also be made to the experimentalteachings and underlying references provided in the underlyingprovisional application, including the Appendices that form part of theprovisional application. Embodiments discussed in the Appendices are notintended, in any way, to be limiting to the overall technicaldisclosure, or to any part of the claimed invention unless specificallynoted.

The Appendices of the underlying Provisional Application are herebyfully incorporated by reference for their general and specificteachings. Appendix A entitled “Precise 3D scan-free multiple-particletracking over large axial ranges with Tetrapod point spread functions”,Appendix B entitled “Precise 3D scan-free multiple-particle trackingover large axial ranges with Tetrapod point spread functions”, AppendixC entitled “Appendix C”, and Appendix D entitled “Tetrapod Phase MaskMicroscopy for high precision three-dimensional position estimation overa large, customizable depth range. Consistent with embodiments of thepresent disclosure, Appendices A, B, and D describe and show examples ofoptical apparatuses and use of the optical apparatuses to localizeobjects in three dimensions. Appendix C shows examples of localizationof objects in three dimensions using an optical apparatus in accordancewith various embodiments.

What is claimed is:
 1. An apparatus comprising: a phase mask configuredand arranged with optics in an optical path to modify a shape of light,passed from an object, wherein the shape modification characterizes thelight as having two lobes with a lateral distance that changes along aline, having a first orientation, as a function of an axial proximity ofthe object to a focal plane, and with the line having a differentorientation depending on whether the object is above or below the focalplane; and circuitry configured and arranged to generate athree-dimensional image from light by using the modified shape toprovide depth-based characteristics of the object.